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unidimensional  will  result  in  fewer  rating  errors  of  inconsistency  was 
tested  using  a Mixed  Standard  Rating  Scale  developed  for  police  super- 
visory personnel.  Two  measures  of  un i dimens iona I i ty  were  used.  The 
correlations  between  the  indices  of  unidimensionality  and  rating  incon-  - 
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Blanz  (Note  1)  introduced  a new  multidimensional  rating  scale  for- 
mat called  the  Mixed  Standard  Rating  Scale  (MSS).  Since  its  introduc- 
tion it  has  been  researched  by  Blanz  and  Chisel li  (1972),  Arvey  and 
Hoyle  (197^),  Saal  and  Landy  (1977)  and  Finley,  Osburn,  Dubin,  and  Jen- 
erret  (1978).  Claims  of  low  levels  of  halo  and  leniency  have  been 
made  with  the  use  of  the  scale  by  Blanz  and  Chisel  I i (1972)  and  Saal 
and  Landy  (1977).  As  with  a behavioral ly  anchored  rating  scale  (BARS), 
performance  dimensions  are  identified  and  behavioral  anchors  are  devel- 
oped to  represent  high,  medium  and  low  levels  of  each  of  the  dimensions. 
All  of  these  anchors  are  then  mixed  and  presented  to  the  rater  in  a 
random  order.  The  rater  is  asked  to  indicate  whether  the  ratee  is 
"better  than,"  "the  same  as,"  or  "worse  than"  each  of  the  anchors.  After 
all  of  the  ratings  are  completed,  the  pattern  of  responses  made  by  a 
rater  on  the  three  anchors  of  each  dimension  are  examined  and  assigned 
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a score  according  to  a system  proposed  by  Blanz  and  Ghiselli  (1972). 

One  of  the  advantages  of  using  this  format  is  that  the  scoring  is  not 
obvious  to  the  rater.  The  rater  is  thus  presumably  forced  to  read  each 
of  the  anchors  rather  than  ignoring  them  and  just  marking  a point  along 
a vertical  continuum.  A MSS  was  developed  in  the  present  study  for  the 
evaluation  of  police  supervisory  performance. 

V/ith  the  use  of  MSS's  a new  type  of  rating  error  can  be  detected 
called  the  inconsistency  error.  By  examining  a rater's  three  responses 
to  the  anchors  of  a performance  dimension,  it  is  possbile  to  detect 
logical  inconsistencies.  For  example,  in  Table  1 the  anchors  which  we 
used  to  represent  high,  medium,  and  low  levels  of  the  dimension  of 
Report  Writing  are  illustrated.  If  an  individual  were  to  be  rated  by 
his  supervisor  as  better  than  the  anchor  representing  a high  level  of 
report  writing  then  it  would  be  an  error  of  inconsistency  for  that  in- 
dividual to  also  be  rated  by  the  supervisor  as  worse  than  the  medium 
or  low  anchors  of  report  writing.  This  type  of  analysis  is  similar  to 
what  has  been  referred  to  in  the  scaling  literature  as  a Guttman  scalo- 
gram  analysis  (cf.  Edwards,  1957).  Several  researchers  have  employed  MSS 
formats  for  the  purpose  of  evaluating  the  scalability  of  their  BARSs 
(Arvey  6 Hoyle,  197^;  Bernardin,  LaShells,  Smith,  6 Alvares,  1976;  and 
Kafry,  Zedeck,  & Jacobs,  1976).  An  examination  of  the  ratings  made  with 
our  scale  revealed  such  a large  number  of  these  inconsistencies  that 
further  explanation  appeared  appropriate. 

In  examining  errors  of  i neons i s tency,  counts  can  be  made  of  the  number 
of  inconsistencies  for  the  ratings  made  of  each  ratee,  the  ratings  made 
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Table  1 


High 


Med i um 


Low 


The  Set  of  Anchors  Used  for  the  Dimension 
Of  Report  Wri ting 

Writes  reports  that  accurately  describe  the  situation. 

Writes  reports  which  describe  events  in  chronological  order. 
Writes  reports  in  a clear  and  understandable  fashion. 
Carefully  separates  opinion  from  fact  in  written  reports. 

Includes  trivial  information  in  written  reports. 

Writes  reports  containing  incomplete  sentences  or  fragments. 

Stretches  the  truth  in  reporting  what  occurred. 

Leaves  out  important  details  in  reports. 


it 


i*. 

by  each  rater,  and  the  ratings  tnade  on  each  rating  scale.  Thus  ratees 
may  be  difficult  to  rate  consistently,  raters  may  rate  inconsistently 
or  scales  may  be  difficult  to  use  consistently.  An  examination  of  the 
large  number  of  inconsistencies  obtained  with  our  scales  indicated  that 
the  errors  were  not  limited  to  any  particular  subset  of  ratees  or  raters. 

Thus,  it  was  decided  to  explore  the  possibility  that  rating  errors  of 
inconsistency  may  be  due  to  a property  of  the  rating  scale  itself. 

Ideally,  the  behavioral  anchors  used  to  define  a performance  dimen- 
sion should  modally  exemplify  that  performance  dimension  and  should  have 
a high  intuitive  loading  on  it.  It  was  reasoned  that  if  the  rater  is 
unclear  as  to  which  performance  dimension  a particular  anchor  represents, 
or  if  the  three  anchors  of  a performance  dimension  appear  to  be  repre- 
senting different  dimensions  of  performance  rather  than  the  same  dimension, 
errors  of  inconsistency  may  occur.  Therefore,  the  property  of  the  MSS 
which  we  examined  was  the  unid imens ional i ty  of  the  rating  scales. 

It  was  hypothesized  that  there  is  an  inverse  relationship  between 
the  unidimensionality  of  the  anchors  which  comprise  a rating  scale  and 
the  number  of  errors  of  inconsistency  which  raters  make  on  a rating  scale, 
such  that  there  arc  fewer  errors  made  on  more  un i d imens iona 1 scales. 
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METHOD 


Job  analysis,  scale  construction,  and  scale  evaluation  were  carried 
out  as  one  part  of  a large  performance  appraisal  project  in  a mid-Atlantic 
police  department. 
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Job  Analysis 

The  job  analysis  stage  involved  defining  the  content  universe,  iden- 
tifying performance  dimensions,  and  assigning  scale  values  to  the  items. 
The  content  universe  was  defined  through  interviews  with  members  from  all 
levels  of  the  organization.  Each  interviewer  developed  a large  pool  of 
behavioral  items  representing  specific  behaviors  of  police  supervisory 
activity.  I terns  were  then  screened  by  both  the  research  team  and  members 
of  the  organization  for  ambiguity  and  job  relevance. 

To  identify  performance  dimensions,  the  edited  pool  of  items  was  sub- 
jected to  a qualitative  cluster  analysis  (Campbell,  Dunnette,  Arvey  and 
Hellervik,  1973).  That  is,  the  items  were  sorted  into  what  appeared  to 
be  homogeneous  categories.  The  resulting  performance  dimensions  were 
then  labelled  and  discussed  with  members  from  the  police  department.  As 
a result  of  this  process,  the  following  ten  performance  dimensions  were 
agreed  upon:  Job  Knowledge,  Teamwork,  Judgment  and  Adaptability,  Report 
Writing,  Dealing  with  the  Public,  Initiative  and  Dedication,  Personal 
Appearance  and  Demeanor,  Consideration,  Structure,  and  Rating. 

Scaling  of  the  items  was  done  by  87  police  officers  representing  all 
levels  of  the  organization.  They  were  each  asked  to  judge  each  of  the 
items  on  a five  point  scale  representing  a continuum  of  effectiveness- 
ineffectiveness.  Mean  scale  values  and  standard  deviations  of  these  judg- 


ments were  then  computed.  I terns  with  standard  deviations  greater  than 


1.5  or  to  which  greater  than  25%  of  the  sample  indicated  were  inapplicable 


to  the  jobs  of  sergeant  or  lieutenant  were  deleted  from  the  item  pool 
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Scale  Construction 

As  previously  discussed,  the  rating  instrument  developed  was  a MSS. 

To  construct  the  scale,  the  pool  of  items  for  each  performance  dimension 
was  divided  into  groups  of  high,  medium,  and  low  scale  values.  The  num- 
ber of  items  chosen  to  represent  each  anchor  varied  from  two  to  five. 
Groups  of  items  rather  than  single  items  were  used  as  anchors  so  that 
raters  would  be  better  able  to  get  a clear  idea  of  the  dimension  and 
level  of  effectiveness  to  which  each  anchor  referred.  I terns  were  not 
included  as  part  of  an  anchor  where  the  overlap  in  scale  values  with  an 
item  anchoring  an  adjacent  level  was  within  two  standard  errors  of  the 
mean.  The  resulting  30  anchors  (three  for  each  of  the  ten  performance 
dimensions)  were  arranged  randomly  on  the  final  rating  form.  The  rater's 
task  was  to  indicate  whether  the  ratee  was  better  than,  the  same  as,  or 
not  likely  to  be  as  good  as  each  of  the  anchors  by  placing  a mark  on  a 
computerized  answer  sheet. 

Sixty-nine  sergeants  were  rated  by  35  lieutenants  and  25  lieutenants 
were  rated  by  1 3 of  their  superiors.  Both  sergeants  and  lieutenants  were 
rated  by  only  one  of  their  superiors.  Several  superiors  rated  more  than 
one  subordinate. 

Scale  Evaluation 

Scale  evaluation  involved  assessing  the  un i dimens ional i ty  of  each  of 
the  separate  rating  scales,  counting  the  number  of  rating  inconsistencies 
made  on  each  scale,  and  relating  these  two  measures. 

In  order  to  assess  the  unidimens ional i ty  of  the  scales  separate  prin- 
cipal component  analyses  (PCA)  were  conducted  for  each  of  the  ten  rating 
dimensions.  Principal  component  analysis  is  a statistical  technique  for 
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describing  the  distribution  of  variance  in  a set  of  variables.  The  new 
variables  or  components  can  be  considered  as  providing  a description  of 
the  structure  of  the  original  set  of  variables  (Harris,  1975,  p.  23). 

The  principal  components  thus  describe  the  dimensionality  of  the  set  of 
variables.  These  analyses  were  conducted  using  as  input  the  effective- 
ness judgments  of  all  of  the  items  used  to  represent  a performance  dimen- 
sion. For  example,  for  the  report  writing  dimension  shown  in  Table  1, 
the  effectiveness  Judgments  of  all  eight  items  in  the  scale  were  used  in 
the  PCA. 

Two  statistics  were  computed  from  the  information  provided  by  these 
PCAs  to  serve  as  indices  of  the  unidimensionality  of  the  rating  dimensions 
The  proportion  of  variance  accounted  for  by  the  first  principal  component 
was  used  as  one  index  of  unidimensional ity  (Index  I).  The  proportion  of 
variance  accounted  for  by  the  principal  component  rather  than  the  actual 
principal  component  values  themselves  were  used  in  order  to  make  the  diff- 
erent PCAs  more  comparable  due  to  the  unequal  number  of  items  comprising 
each  scale.  The  second  index  created  was  the  ratio  of  the  first  princi- 
pal component  to  the  second  principal  component  (Index  2).  This  index  was 
designed  to  explore  the  possibility  of  a rival  major  dimension  in  the  data 
For  both  Index  1 and  Index  2,  the  greater  the  value  of  the  index,  the 
greater  the  evidence  for  unidimensionality  of  the  scale. 

There  are  27  possible  rating  combinations  which  a rater  may  use  to 
describe  a ra tee's  status  on  any  given  rating  scale  (Saal  and  Landy,  1977, 
p.  2A).  These  combinations  are  listed  in  Table  2.  Note  that  seven  of 
these  rating  combinations  are  totally  consistent  while  the  remaining  20 


MSS  Error  Counting  System 


Anchors 


Consistent  Combinations 


Inconsistent  Combinations 
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contain  some  degree  of  inconsistency.  Each  time  a rater  described  a ratee 
inconsistently  on  a particular  rating  dimension  this  was  counted  as  an 
error.  A count  was  kept  of  the  number  of  errors  made  for  each  of  the  ten 
rating  dimensions. 

The  major  hypothesis  of  the  study  was  tested  by  correlating  this 
error  count  with  the  two  unidimensionality  indices  across  the  ten  rating 
dimensions. 


I 

I 
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RESULTS 

Table  3 lists  the  two  indices  of  unidimensionality  for  each  of  the 
ten  rating  dimensions.  The  correlation  between  these  two  indices  is  .78. 
The  strength  and  direction  of  this  relationship  was  expected  due  to  the 
non- independence  of  the  two  indices. 

The  ratings  of  both  sergeants  and  lieutenants  were  characterized  by 
a large  number  of  inconsistency  errors.  A total  of  95^  of  the  ratings 
were  inconsistent  (95.6%  for  sergeants  and  93*2%  for  lieutenants).  The 
proportion  of  inconsistent  ratings  obtained  within  each  of  the  ten  rating 
dimensions  for  the  ratings  of  sergeants  and  lieutenants  are  presented, in 
Table  1*.  These  were  computed  by  dividing  the  number  of  errors  made  on  each 
dimension  by  the  total  number  of  ratings  made  on  that  dimension. 

The  intercorrelations  between  the  two  indices  of  unidimensionality 
and  the  error  counts  for  sergeants  and  lieutenants  are  shown  in  Table  5* 

All  correlations  are  significant  and  in  the  predicted  direction.  A nega- 
tive correlation  denotes  the  fact  that  the  higher  the  index  of  unidimen- 
sionality, the  fewer  the  number  of  inconsistencies. 
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Table  3 


Unidimensional ity  Indices  for 

Each  of  the  Ten 

Rating  Dimensions 

Rating  Dimension 

Index  1 

Index  2 

Report  Writing 

.470 

.151 

Initiative  and  Dedication 

.438 

.203 

Consideration 

.369 

.165 

Rating 

.354 

.138 

Dealing  with  the  Public 

.337 

.145 

Personal  Appearance  and  Demeanor 

.315 

.128 

Structure 

.304 

.139 

Job  Knowledge 

.296 

.131 

Judgment  and  Adaptability 

.288 

.115 

Teamwork 

.268 

.107 
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Table  k 

Proportion  of  Rating  Errors  of  the  Ratings  of  Sergeants  and  Lieutenants 

For  Each  Rating  Dimension 


Rating  Dimension 

Sergeants 

Lieutenants 

Job  Knowledge 

.97 

.96 

Teamwork 

l.OO 

.96 

Judgment  and  Adaptability 

1.00 

1.00 

Report  Writing 

.86 

.80 

Dealing  with  the  Public 

1.00 

.92 

Initiative  and  Dedication 

.80 

.84 

Personal  Appearance  and  Demeanor 

.99 

.92 

Consideration 

.97 

1.00 

Structure 

.97 

.92 

Rating 

1.00 

1.00 
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DISCUSSION 

The  major  hypothesis  of  the  study  was  confirmed.  It  was  found  that 
the  consistency  with  which  raters  are  able  to  use  multidimensional  be- 
havior-specific rating  scales  is  positively  related  to  the  unidimension- 
ality of  those  scales. 

In  considering  this  result  it  should  be  kept  in  mind  that  the  statis- 
tical assessments  of  unidimensionality  are  based  upon  judgments  of  stimuli 
made  during  the  developmental  stages  of  the  instrument  while  the  measure 
of  rating  inconsistency  is  based  upon  the  data  obtained  from  actual  ratings 
of  people  made  with  the  scales.  Thus,  the  result  represents  a relation- 
ship between  two  very  different  types  of  measurements. 

Two  major  features  of  the  results  should  be  discussed.  First, 

it  should  be  noted  that  the  study  was  concerned  with  the  relative  levels 

/ 

of  unidimensionality  of  the  separate  scales  rather  than  absolute  levels. 
Just  how  acceptable  th<!  levels  of  unidimensionality  are  was  not  a prime 
concern.  The  study  demonstrates  that  the  relative  degree  of  unidimen- 
sionality of  the  scales  is  related  to  the  relative  consistency  of  raters' 
use  of  the  scales. 

Second,  the  ratings  of  the  present  study  contained  a large  number  of 
inconsistencies.  Ninety-five  percent  of  the  ratings  were  inconsistent. 
Several  possible  explanations  can  be  offered. 

Based  upon  the  major  result  of  the  study,  the  most  obvious  explana- 
tion is  the  level  of  multidimensionality  of  the  scales.  However,  just 
how  mul tidimentional  the  scales  are  is  difficult  to  judge  without  com- 
parative data. 


\k. 

Post  hoc  analyses  suggest  that  the  use  of  groups  of  items  as  anchors 
may  have  contributed  to  the  large  number  of  inconsistent  ratings.  It  was 
hoped  that  the  use  of  groups  of  items  would  serve  to  better  define  the 
anchors  than  would  the  presentation  of  single  items.  However,  the  inclusion 
of  a number  of  items  to  anchor  each  level  of  a dimension  may  have  increased 
the  difficulty  of  using  the  rating  scales.  The  correlations  between  the 
total  number  of  items  used  in  the  anchors  of  a scale  and  the  number  of 
errors  of  inconsistency  in  rating  with  that  scale  are  significant 
(£  = .63,  £<.05  for  sergeants,  j-  = .78,  £<.01  for  1 ieutenants , two- 

tailed  tests).  Furthermore,  the  number  of  items  in  a scale  may  contri- 
bute to  the  dimensionality  of  each  scale.  A post  hoc  analysis  of  the 
correlation  between  the  number  of  items  anchoring  a scale  and  the  uni- 
dimensionality indices  suggests  a possible  relationship  (£  = .62,  £<.10 
for  Index  1;  £-  *^7,  £<.20  for  index  2,  two-tailed  tests). 

Another  possible  reason  for  the  large  number  of  inconsistencies 
is  the  fact  that  behavioral  items  rather  than  general  trait-like  items 
were  used  for  the  anchors.  Concern  for  the  appropriateness  of  behavicr- 
specific  anchors  in  MSS^  was  raised  by  Finley,  Osburn,  Dubin  and  Jeanneret 
(1977).  They  stated  that  the  MSS  responses  "are  not  appropriate  for 
behavioral ly  specific  anchors  presented  to  the  rater  out  of  scale  context." 
(p.  660)  Their  concern  appears  to  be  that  a specific  anchor  by  itself 
does  not  reveal  an  adequate  amount  of  information  for  the  rater  to  be 
able  to  identify  the  rating  dimension  to  which  the  anchor  applies. 

The  question  of  how  non-unidimensional i ty  might  be  related  to  errors 
of  inconsistency  in  MSS^  may  be  explained  by  examining  in  detail  the 
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rating  process  itself.  Ratings  on  a MSS  presumably  require  a series  of 
several  cognitive  processes.  After  reading  a behavioral  anchor,  the  rater 
must  identify  both  the  performance  dimension  and  the  level  of  effective- 
ness which  it  represents  by  isolating  the  performance  dimension  from  other 
performance  dimensions,  recalling  i 1 lustrations  of  the  ratee's  behavior 
relevant  Jto  the  performance  dimension,  and  then  finally,  comparing  these 
recalled  behaviors  to  the  anchor  in  terms  of  level  of  effectiveness.  It 
is  the  outcome  of  this  type  of  complex  process  which  leads  to  the  rater's 
decision  to  respond  better  than,  same  as,  or  not  likely  to  be  as  good  as. 

Following  through  with  this  view  of  the  rating  process,  if  an  anchor 
is  multidimensional,  it  is  likely  to  have  multiple  meanings  for  the  rater. 
Thus,  the  complexity  of  the  cognitive  demands  of  the  task  (Schneier,  1977) 
will  be  greatly  increased.  It  would  then  be  a difficult  task  for  the  rater 
to  identify  and  isolate  either  the  dimension  or  the  level  of  effectiveness 
to  which  the  anchor  applies  leading  to  difficulties  in  recalling  the  re- 
levant ratee  behaviors  and  in  comparing  these  behaviors  to  the  anchor. 

If  the  separate  anchors  of  a scale  appear  to  the  rater  to  represent  differ- 
ent performance  dimensions,  inconsistency  errors  may  occur  because  the  rater 
considers  different  ratee  behaviors  relevant  when  comparing  the  ratee  to 
each  of  the  anchors. 

If  multidimensionality  leads  to  an  increase  in  the  cognitive  complexity 
of  the  task,  the  results  of  this  study  may  be  applicable  to  other  multi- 
dimensional behavior-specific  rating  formats  such  as  the  BARS.  The  series 
of  cognitive  processes  which  characterize  the  MSS  and  BARS  tasks  may  not 
differ  greatly.  Ideally,  to  use  a BARS  the  rater  reads  all  of  the 
anchors,  identifies  the  appropriate  performance  dimension,  recalls  the 
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relevant  ratee  behaviors,  compares  these  behaviors  to  each  of  the  anchors, 
and  then  makes  the  rating  by  deciding  which  of  the  anchors  best  matches 
these  ratee  behaviors.  The  major  difference  between  the  BARS  and  MSS 
tasks  is  that  with  the  BARS  the  rater  is  provided  with  more  information. 

All  of  the  anchors  are  presented  at  once  with  each  anchor’s  physical  loca- 
tion on  the  scale  indicating  its  level  of  effectiveness.  Presumably  the 
rater  is  thus  better  able  to  understand  what  performance  dimension  the 
scale  represents  and  what  level  of  effectiveness  is  represented  by  each 
anchor. 

As  with  the  MSS,  however,  non-unidimensional i ty  of  the  anchors  on  a 
given  scale  may  influence  the  rating  task.  If  the  scale  as  a whole  does 
not  appear  to  represent  a unidimensional  continuum,  then  the  rater's 
task  of  referring  to  relevant  ratee  behaviors  and  comparing  these  recalled 
behaviors  to  the  anchors  may  be  difficult.  The  rater  may  be  forced  to 
base  his  or  her  decision  on  a general  overall  impression  or  to  make  in- 
valid compromises  between  different  anchors.  The  BARS  procedure  provides 
no  way  to  take  note  of  inconsistency  errors.  One  possible  solution  to 
this  problem  has  been  to  pilot  the  BARS  using  a MSS  format  as  was  done 
by  Arvey  and  Hoyle  (197^). 

The  major  implication  of  the  study  is  that  since  rater's  errors  of 
inconsistency  are  related  to  properties  of  the  rating  scale,  researchers 
interested  in  constructing  multidimensional  behavior-specific  scales  should 
be  checking  their  rating  scales  for  unidimensional i ty.  One  possible 
statistical  technique  for  doing  so  is  presented. 
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Another  statistical  approach  would  be  to  use  factor  analytic  tech- 
niques. Dickinson  and  Tice  (1977)  used  a multiple-group  factor  analysis 
of  the  scale  values  of  their  items  to  identify  those  items  which  loaded 
unequivocally  on  a single  performance  dimension.  Through  this  approach 
they  were  able  to  reduce  the  factorial  complexity  of  their  rating  dimen- 
sions. Evidence  of  increased  discriminant  validity  of  the  scales  was  used 
to  substantiate  this  claim. 

Another  approach  would  be  to  increase  the  retranslation  criterion 
used  during  scale  construction  (Smith  6 Kendall,  1963) • This  merely 
Involves  the  setting  of  a high  level  of  agreement  among  the  scale  construc- 
tors concerning  which  behaviors  belong  to  each  of  the  performance  dimen- 


sions. However,  Kafry,  Zedeck,  and  Jacobs  (1976)  found  that  varying  this 
had  no  effect  on  the  scalability  of  their  scales. 

A third  approach  to  this  problem  which  may  be  suggested  would  be  to 
more  closely  examine  the  rating  process  itself.  Introspective  reports  of 
raters  might  provide  useful  information  concerning  the  unidimensionality 
of  the  scales. 
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